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Abstract 

Natural images in the colour space YUV have been ob- 
served to have a non-Gaussian, heavy tailed distribution 
(called 'sparse') when the filter 

7 (C/)(r) = U(r) - £ w(Y)„U(b), 

seiV(r) 

is applied to the chromacity channel U (and equivalently 
to V), where w is a weighting function constructed from 
the intensity component Y [1]. 

In this paper we develop Bayesian analysis of the col- 
orization problem using the filter response as a regulariza- 
tion term to arrive at a non-convex optimization problem. 
This problem is convexified using L 1 optimization which 
often gives the same results for sparse signals [ ] . It is ob- 
served that L 1 optimization, in many cases, over-performs 
the famous colorization algorithm by Levin et al [3]. 

1 Introduction 

Colorization of natural images has been a long standing 
problem in image processing. The initial process was in- 
vented by Wilson Markle and Brian Hunt and first used in 
1970 to add colour to monochrome footage of the moon 
from the Apollo mission. One of the main drawbacks of 
this and subsequent techniques has always been the labour 
intensity involved and hence the associated costs. This is 
due to the time-consuming segmentation and tracking of 



objects in images plus the assignment of colours that all 
require substantial user involvement. 

However, a recent state of the art method has been pro- 
posed by Levin et al in the classical paper [ ] where col- 
orization is performed by optimization. In the mentioned 
work a fundamental hypothesis is that areas of similar lu- 
minance should have similar colours. This, together with 
additional colour scribbles placed on the interior regions 
of objects in the gray image, is used to propagate colour to 
the rest of the image by minimisation of a quadratic cost 
function. The result is a visually pleasing image with a 
reduction in user input. 

A number of recent advancements have since been 
made to improve the quality and efficiency of the col- 
orization process. These works can roughly be divided 
into scribble based and example-based colorization. Two 
highlighted cases are the works presented in [4] and [5]. 
In the former paper a computationally simple, yet effec- 
tive, approach is presented which works very fast and 
can be conveniently used 'on the fly', permitting the user 
to promptly get the desired results after providing a set 
of chrominance scribbles. The latter paper develops the 
method of transferring colour from a segmented example 
image, and uses the method in [ ] to produce the finished 
colorized image. This method has the advantage of not 
having to rely upon the user's skill or experience in choos- 
ing suitable colours for a convincing colorization. 

Partially inspired by the work in [ ] we study the col- 
orization problem using Bayesian analysis, where we use 
the response of the filter observed in [1] as a regulariza- 
tion term; 
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7 (C/)(r) = U(v) - ™( Y )r*U(s), (1) 

seN(r) 

where r represents a two dimensional point, N(r) a 
neighborhood (e.g. 3x3 window) of points around r, and 
w(Y) rs a weighting function. The filter is applied to the 
chromacity channel U (and equivalently to V) of individ- 
ual natural images in the colour space YUV. This par- 
ticular space is chosen as it allows the decoupling of the 
luminance and colour components of an image. We note 
here that since the U and V elements are similar, our work 
is only explained for the U component where analysis of 
the V component is obtained by substitution. 

The filter response can be modelled by a Generalised 
Gaussian Distribution (GGD), 

J a (x) = ^e^" , (2) 

where Z is a normalising constant so that the integral of 
J a (x) is 1, s the scale parameter and a the shape param- 
eter. The GGD gives a Gaussian or Laplacian distribution 
when a = 2 or 1, respectively. When a < 1 we have a 
'heavy tailed' distribution which we call 'sparse'. 

Figure 1 shows a typical example (with the vertical 
axis on a log scale) of the histogram of pixel intensity 
that is observed after filtering a natural colour image. 
Fitted to the data is a GGD that takes the form of a 
sparse distribution function. We also overlay on the fig- 
ure the classical parabola shaped Gaussian distribution 
which clearly shows the difference in the tails between the 
two. Such differences highlight the importance of choos- 
ing the correct distribution as a priori knowledge when 
using Bayesian analysis. 

Our key contribution in this paper is tackling the col- 
orization problem using Bayesian analysis with the sparse 
distribution property of the filter response on natural im- 
ages acting as a regularization term. We arrive at a non- 
convex optimisation problem taking the form of a GGD 
with shape parameter a < 1. Interestingly, our regular- 
ization term indicates that the quadratic cost function (i.e. 
using a = 2 in the GGD model) minimised in [ ] is in fact 
a Gaussian approximation to the true objective function. 



2 Bayesian Analysis of the Coloriza- 
tion Problem 

Our colorization problem is posed as follows: We 
are given a gray level natural image Y and additional 
coloured markings U Q on S, where S denotes a subset 
of pixels of the image. We would like find to a U on the 
whole image s.t. 

(cl) U\ 8 = U , 

(c2) and the resulting colour image looks natural. 

Formally we have the following: For any A let us de- 
note by Py(A) the conditional probability P(A\Y). Then 
we wish to maximise 

Vy(U\U ). (3) 
Applying B ayes' formula results in maximising 

Vy(U \U)V y (U), (4) 
or equivalently to find 

argmaxjjPy{U\ (5) 

under condition (cl). 

Remark: If we want to keep U Q exactly then Py (Uq\U) 
islifU\s = U and otherwise. If we assume that we 
can have error in U then Py(U \U) can be modelled as 
a Gaussian distribution for U\s — U Q . 

We model Py(U) by analysing the filter response in 
[1]. More precisely we use the following expression: 

P Y (U) oce- £ bi*u\" 

where gi is the filter operating on the z'th pixel in the im- 
age. Taking logs leads to an equivalent minimisation ob- 
jective, 

argminu ^ \( 9i • U)\ a with U\ s = U Q . (6) 

i 

With a = 2, we arrive at the same optimization prob- 
lem solved by Levin et al in [3]. Moreover, this shows 
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that their model gives colorization with Gaussian behav- 
ior of the response gi*U. However, for all natural images 
that we have studied, we found that < a < 2 but rarely 
a > 1 implying the distribution of the filter response to 
generally be sparse, thus arriving at the correct optimiza- 
tion function. 

3 Optimization using the L 1 norm 

Solving 6 for a < 1 leads to a non-convex optimization 
problem that unlike least squares regression has no ex- 
plicit formula for the solution. We defer the case a < 1 
for future work and instead convexify the problem using 
L 1 optimization which often gives the same results for 
sparse signals, especially if the size of |5| is small com- 
pared to the size of the image [ ]. Integrating the con- 
straints U\s = U into the objective function this prob- 
lem can be reformulated as an unconstrained one where 
we seek to minimise, 

i ieS 

with i running over all image pixels and S the set of pixels 
that have been marked by colour. This can be constructed 
into the vectorial form 

\\AU-b\U (8) 

where 1 1 • | |i represents the L 1 norm and the i'th row of 
the sparse matrix A corresponds to the filter response of 
the z'th pixel in the image, with b constructed such that 7 
agrees with 8. 

Using slack variables Vi and 8 can be constructed as 
a linear program, 

s.t. AU + v-fjL = b, (9) 
Vi,lii,U(i),bi > 

The idea here is to find the smallest pairwise addition v { + 
Hi, such that their difference is equal to b(i) —A^U. This 
occurs precisely when one of the vi or \i% are zero an d the 
other equal to b(i) — A^U, handling both the positive 
and negative cases. 
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We use the linear programming package 'LIPSOL' [6] 
available through high-level programming environments, 
Scilab and Matlab, to solve the problems. Images in the 
region of 250x250 pixels take a few minutes to colorize 
and hence our method is slower than the solvers used in 
[ ] . However, our goal here is not to efficiently solve such 
problems, but only to state the correct optimization prob- 
lem and to show that when such a problem is solved, the 
resulting colorized image is of a higher quality. 

The results shown in the figures compare the quality 
of the colorization using the L 1 optimisation against the 
approach of Levin et al. Marking large regions of pix- 
els gives similar results, however, using a much smaller 
set of marked pixels highlights the difference between the 
two methods. We note here that since we are only con- 
cerned with the correct propagation of colour, and not the 
choosing of colour, we use the original colour channels of 
the images for marking colour points on a converted gray 
image. 

Figure 2 shows an example image from the paper by 
Levin et al. We colorize the same image but using a much 
sparser set of marked pixels placed arbitrarily in the im- 
age regions, (c) shows the improvement in colorization 
using L 1 optimization where we observe a sharper result 
and not an oversmoothed output as usually is the case for 
assuming a Gaussian prior. 

Figure 3 shows an example of a colorized image. Re- 
quiring a relatively few set of marked pixels our method 
shows good results compared to that of Levin et al's which 
incorrectly colorizes the red balloon in the centre of the 
picture as purple. We also observe more vibrancy in the 
colours in (c) over (d), particularly the stripes on the t- 
shirt. 

Our results are still a working progress and we are cur- 
rently improving the efficiency and quality of the col- 
orization. In future we hope to further improve the sharp- 
ness and accuracy of the output images whilst signifi- 
cantly reducing the user marked pixels required in the op- 
timization scheme. 
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5 Summary 



The problem of colorizing natural images is tackled us- 
ing Bayesian analysis where we use the sparse distribu- 
tion property of the filter response observed in [ ] as a 
regularization term. We arrive at a cost function taking 
the form of a GGD with shape parameter a < 1 resulting 
in a non-convex optimisation problem. Our regularization 
term indicates that the quadratic cost function (i.e. using 
a = 2) minimised in [ ] is in fact a Gaussian approxima- 
tion to the true objective function. 

We convexify the non-convex problem using L 1 opti- 
mization which is often found to give the same results 
for sparse signals. This problem is formulated as a linear 
program that can be solved using any standard method; 
we chose to use the software package 'LIPSOL' available 
through the programming environments Scilab and Mat- 
lab. 

It is observed that L 1 optimization, in many cases, over- 
performs the famous colorization algorithm by Levin et al 
[ ], although at present solving for an equivalent sized 
image takes longer. However, our aim here is not find the 
most efficient solvers for such problems, but only to state 
the correct formulation of the colorization problem. 
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Figure 2: Colorization example, (a) The gray image marked by a 
sparse set of colour pixels; (b) the original image for reference; (c) col- 
orization using L 1 optimization; (d) Levin et al; (e) and (f) closer in- 
spection of (c) and (d) respectively. Here we have an original image 
from the paper by Levin et al, this time colorized with a much sparser 
set of pixels placed arbitrarily on the image. We observe more vibrancy 
in the colours in (c) against the 'washed out' look of the colorization 
in (d). We also have a sharper result observed upon closer inspection 
and not an oversmoothed output as usually is the case for assuming a 
Gaussian prior. 
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Figure 3: Colorization example. Here we have a comparison of the 
visual quality produced by L 1 optimization against the technique used 
by Levin et al. (a) is an example gray image marked by a sparse set 
of coloured pixels arbitrarily placed; (b) the original colour image for 
reference; (c) shows colorization using L 1 optimization; (d) method of 
Levin et al. We observe a more accurate colorization in (c) e.g. the red 
balloon in the centre of the image is colorized correctly as opposed to 
the purple colorization in (d). We also observe more vibrant and sharper 
colours in (c), particularly on the t-shirt. 
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